<html>
<head><meta charset="utf-8"><title>ask llvm to generate obj in memory? · t-compiler/wg-self-profile · Zulip Chat Archive</title></head>
<h2>Stream: <a href="https://rust-lang.github.io/zulip_archive/stream/187831-t-compiler/wg-self-profile/index.html">t-compiler/wg-self-profile</a></h2>
<h3>Topic: <a href="https://rust-lang.github.io/zulip_archive/stream/187831-t-compiler/wg-self-profile/topic/ask.20llvm.20to.20generate.20obj.20in.20memory.3F.html">ask llvm to generate obj in memory?</a></h3>

<hr>

<base href="https://rust-lang.zulipchat.com">

<head><link href="https://rust-lang.github.io/zulip_archive/style.css" rel="stylesheet"></head>

<a name="189100065"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187831-t-compiler/wg-self-profile/topic/ask%20llvm%20to%20generate%20obj%20in%20memory%3F/near/189100065" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Charles Lew <a href="https://rust-lang.github.io/zulip_archive/stream/187831-t-compiler/wg-self-profile/topic/ask.20llvm.20to.20generate.20obj.20in.20memory.3F.html#189100065">(Feb 26 2020 at 10:02)</a>:</h4>
<p>Hello, after reading the blog post, it seems there's indeed a large fraction of time consumed in waiting for llvm operations complete, especially generating obj files on disk. This makes me wonder that whether asking llvm to generate such data in memory and write them to disk only when absolutely necessary (then in batch) will improve compiling performance.</p>



<a name="189109402"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187831-t-compiler/wg-self-profile/topic/ask%20llvm%20to%20generate%20obj%20in%20memory%3F/near/189109402" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Hanna Kruppe <a href="https://rust-lang.github.io/zulip_archive/stream/187831-t-compiler/wg-self-profile/topic/ask.20llvm.20to.20generate.20obj.20in.20memory.3F.html#189109402">(Feb 26 2020 at 12:27)</a>:</h4>
<p>The <code>LLVM_module_codegen_emit_obj</code> step includes going from LLVM IR to object file contents in the first place, it's not just writing stuff to disk. I don't have numbers but I don't see any reason to expect that the I/O takes any significant fraction of that time, most or all of it should be taken up by real work needed for code generation.</p>
<p>It's also not clear to me if buffering entire files in memory would even help -- even if it would save a roundtrip through the file system (only possible for some output formats), if there's enough free memory to keep the file in memory then the OS might cache it anyway if it gets written to disk &amp; read back.</p>



<a name="189110632"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187831-t-compiler/wg-self-profile/topic/ask%20llvm%20to%20generate%20obj%20in%20memory%3F/near/189110632" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Laurențiu <a href="https://rust-lang.github.io/zulip_archive/stream/187831-t-compiler/wg-self-profile/topic/ask.20llvm.20to.20generate.20obj.20in.20memory.3F.html#189110632">(Feb 26 2020 at 12:47)</a>:</h4>
<p>I've been a little confused by <code>finish_ongoing_codegen</code> -- I don't think it finishes anything, it just waits for codegen to finish on other threads. And if that takes too long, it's not necessarily because of IO.</p>



<a name="189114182"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187831-t-compiler/wg-self-profile/topic/ask%20llvm%20to%20generate%20obj%20in%20memory%3F/near/189114182" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Charles Lew <a href="https://rust-lang.github.io/zulip_archive/stream/187831-t-compiler/wg-self-profile/topic/ask.20llvm.20to.20generate.20obj.20in.20memory.3F.html#189114182">(Feb 26 2020 at 13:39)</a>:</h4>
<p>It seems from (people's experience)[ <a href="https://users.rust-lang.org/t/stylistic-guide-for-speeding-up-compile-time/29592/11?u=crlf0710" target="_blank" title="https://users.rust-lang.org/t/stylistic-guide-for-speeding-up-compile-time/29592/11?u=crlf0710">https://users.rust-lang.org/t/stylistic-guide-for-speeding-up-compile-time/29592/11?u=crlf0710</a> ], in reality disk I/O is currently a major affecting factor for compilation times. However there're other steps that are IO-heavy (like linking etc) so i'm not sure how much of that belongs to the <code>LLVM_module_codegen_emit_obj</code> step.</p>



<a name="189124565"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/187831-t-compiler/wg-self-profile/topic/ask%20llvm%20to%20generate%20obj%20in%20memory%3F/near/189124565" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> bjorn3 <a href="https://rust-lang.github.io/zulip_archive/stream/187831-t-compiler/wg-self-profile/topic/ask.20llvm.20to.20generate.20obj.20in.20memory.3F.html#189124565">(Feb 26 2020 at 15:30)</a>:</h4>
<p>Somebody had a 2x improvement by switching from NFS to an SSD: <a href="https://github.com/bjorn3/rustc_codegen_cranelift/issues/865#issuecomment-574747252" target="_blank" title="https://github.com/bjorn3/rustc_codegen_cranelift/issues/865#issuecomment-574747252">https://github.com/bjorn3/rustc_codegen_cranelift/issues/865#issuecomment-574747252</a></p>



<hr><p>Last updated: Aug 07 2021 at 22:04 UTC</p>
</html>